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ABSTRACT 


This study examined the effects of intervention with a 2 x 2 factor manipulation, identifying the effects of 
evaluation indicators (presentation/construction) and evaluator (self/peer) on students’ goal orientation. Specifically, we 
created five groups (one control group and four experimental groups) and conducted experimental lessons. The learning 
topic was debate training. The participants were 67 first-year students. Analyses focusing on the cross-lagged 
relationships of goal orientation before and after the intervention were performed. The effects of the intervention on goal 
orientation may depend on the individual student’s initial goal orientation. By increasing the level of the intervention, 
students’ initial goal orientation might have a greater effect on the goal orientation that the student exhibited after the 
intervention. The results of this study give guidelines on the learning intervention according to student's initial goal 
orientation. 
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1. INTRODUCTION 


1.1 Goal Orientation 


Goal-achievement theory (Dweck 1986), which accounts for differences in learning behavior in terms of 
what each student’s goals are as they execute tasks, classifies such goals into two categories: learning and 
performance goals. The purpose of the former is to acquire new knowledge and skills through challenging 
activities while that of the latter is to seek positive and avoid negative evaluations. Students with learning 
goals tend to select challenging tasks and persevere even when they encounter failure, regardless of whether 
they are confident in their abilities. Performance-oriented appear similar to students with the learning 
orientation, provided they are confident in their abilities; however, if they lack confidence, they are less 
likely to persevere until they have completed the given tasks. Elliot and Dweck’s (1988) findings support the 
latter statement. Ames and Archer (1987, 1988) note that learning goals have a positive effect on both 
academic achievement and endogenous motivation; learning goals are generally considered superior for 
attaining learning achievement. 

Elliot and Harackiewicz (1996) divided performance goals into a performance-approach goals, in which a 
student tries to outperform others, and performance-avoidance goals, or attempting to avoid performing 
worse than others. Elliot and Church (1997) and Rawsthorne and Elliot (1999), among others, have found 
that performance-approach goals lead to positive effects on endogenous motivation and 
academic performance, but performance-avoidance goals have negative effects, thus demonstrating the 
importance of distinguishing between the approach and avoidance utilities. 


1.2 Stability and Change in Goal Orientation 


These perspectives treat goal orientation either as a personality characteristic— goal orientation is considered 
to be consistent and not to depend on the learning context—or a controllable characteristic—goal orientation 
is induced by the learning environment (DeShon & Gillespie 2005). We take the latter position here and 
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create an intervention intended to change the student’s goal orientation. Fryer and Elliot (2007) and Muis and 
Edwards (2009) examined stability and change in students’ goal orientations. These studies indicate that goal 
orientation is relatively stable, but there is the possibility that it may change in response to certain situations. 
It is important to account for both stability and change in goal orientation (Fryer & Elliot 2007). Changes in 
goal orientation in response to interventions may be affected by the initial goal orientation of the student. 


1.3 Feedback as Change Agent for Goal Orientation 


Winne et al. (2003) intervened in students’ goal orientation using feedback and examined the resulting 
changes in goal orientation. They found that positive feedback resulted in decreases in the 
performance-avoidance goal orientation, and negative feedback resulted in decreases in_ the 
performance-approach goal orientation. However, no increase in the learning-goal orientation was found. 
Geitz et al. (2015) intervened in students’ goal orientation with feedback, using the method indicated by 
Boud and Molloy (2013), and they examined the effects on goal orientation and learning behavior. The 
intervention did not influence goal orientation directly, but it helped goal-oriented students maintain deep 
learning. 

The findings of Winne et al. (2003) made it clear that feedback could play a role in changing goal 
orientation, and Geitz et al. (2015) suggested that the effectiveness of feedback intervention depends on the 
initial goal orientation of the student. We assumed that the effects of changing goal orientation through 
feedback intervention depends on the initial goal orientation of the student. 


1.4 Self-Evaluation / Peer-Evaluation 


In this study, we manipulated two factors. The first was the evaluator. It is important who provides feedback. 
According to Boud and Molly (2013), to create effective feedback, it is necessary for students to be actively 
involved in their own learning and to be agents of their own change. In this study, to increase the 
involvement of students in their learning, we adopted a self-evaluation, in which students provide themselves 
with feedback, and a peer-evaluation, in which students give feedback to each other, in the place of any 
evaluation from teachers. 

Peer-evaluation has been found to improve student motivation (Asghar 2010), and it encourages students 
to draw each other’s attention to what they know and do not know (Topping 2005; Ladyshewsky & Ryan 
2006). Thus, peer-evaluation was expected to be more effective in enhancing student involvement in their 
learning process than self-evaluation was. Peer-evaluation was considered to be the higher level of 
intervention. 


1.5 Presentation / Construction of Evaluation Indicators 


The second factor concerned evaluation indicators. We examined the effects of presenting evaluation 
indicators to students in advance using rubrics, in one manipulation, while in another, we let students make 
rubrics themselves, meaning that they were responsible for identifying indicators. Rubrics are assessment 
tools that articulate specific expectations for assignments by listing the criteria for higher marks, identifying 
what is particularly important in students’ work, and describing levels of quality on a scale from excellent to 
poor (Panadero & Romero 2014). The standard usage is to present students with rubrics constructed by 
teachers, but Anderson (1998) recommends that students be involved in constructing them. In this way, 
students can gain awareness of their own involvement in their learning and can be expected to take tasks 
seriously (Steaven & Levi 2013). The intervention of having students construct evaluation indicators 
themselves was expected to be more effective for enhancing involvement in learning than the intervention in 
which evaluation indicators were presented in advance and represented a higher level of intervention. 


1.6 Purpose 


This study examined the effects of intervention with a 2 x 2 factor manipulation, identifying the effects of 
evaluation indicators (presentation/construction) and evaluator (self/peer) on students’ goal orientation. 
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Specifically, we created five groups: one control group (Control Group) and four experimental groups 
(Presentation-Self Group, Construction-Self Group, Presentation-Peer Group, Construction-Peer Group) and 
conducted experimental lessons. Because the level of intervention for Construction was higher than for 
Presentation, and the level of intervention for Peer was higher than for Self, the relationships of expected 
effects were as follows. 

Control Group < Presentation-Self Group < Construction-Self Group, Presentation-Peer Group 
< Construction-Peer Group 

Two analyses focusing on the cross-lagged relationships of goal orientation before and after the 
intervention were performed, as described below. Analysis | revealed the cross-lagged relationships of each 
group regarding goal orientation using multi-group structural equation modeling (SEM) analysis. Analysis 2 
identified the path-coefficient differences between the control group and each of the four experimental 
groups and clarifies the characteristics of each group by comparison with the control group, which is taken to 
be the baseline. 


2. METHOD 


We intervened in classes held in a public university in 2018. The learning topic was debate training, as the 
student-centered learning was more likely to increase the involvement of students in learning and to change 
their goal orientation. Classes were held five times every other week. The participants were 67 first-year 
students (Control Group: N = 14; Presentation-Self Group: N = 14; Construction-Self Group: N = 13; 
Presentation-Peer Group: N = 13; Construction-Peer Group: N = 13). Each group was subsequently divided 
into three smaller groups (three to five students per group). Table | shows the class procedure. 


Table 1. Class Procedure 


Week 1 *Pre-test (10 min) * Watching an introduction video of debate discussion (60 min) 
‘Evaluation indicators (presentation/construction) (20 min) 

Week 2 Practice for debate (90 min) 

Week 3 Preparing for match up debate (90 min) 

Week 4 


‘Preparing for match up debate (45 min)* Evaluating performance: evaluator (self/peer) (45 min) 


Week 5 *Match up debate (80 min) *Post-test (10 min) 


Pre-test and post-test 

The questionnaire comprised 18 items, which was modified from Mitsunami (2010) translated in 
Japanese from the Achievement Goal Scale developed by Elliot and Church (1997). 

Evaluation indicators (presentation/construction) 

We explained to the students in the four experimental groups that a rubric is a tool for indicating a desired 
goal and its evaluation indicators. Teacher-constructed rubrics were distributed to students of the two 
Presentation Groups. Empty sheets were provided for students of the two Construction Groups. These 
students were asked to carefully consider the goals that they wished to achieve through studying their lessons 
and to fill in the evaluation items and their level. Each group constructed one rubric. 

Evaluator (self/peer) 

We explained to the students in the four experimental groups that the purpose of evaluation is to improve 
learning behaviors to achieve a goal. The students in the two Self Groups conducted self-evaluation, and 
students in the two Peer Groups conducted peer-evaluations, adapting the evaluation indicators of the rubric, 
which was presented or constructed during the first week. 


3. RESULTS 


The responses of 63 subjects who attended lessons during weeks 1, 2, and 5 and responded to all questions in 
the questionnaire were analyzed. The model fit was using by the comparative fit index (CFI) and the root 
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mean square error of approximation (RMSEA). Values above .95 for CFI and below .07 for RMSEA were 
regarded as indicating sufficient fit (Hooper et al. 2008). 


3.1 Confirmatory Factor Analysis 


A confirmatory factor analysis was performed to test the three-factor structure, composed of the learning goal 
(LG), performance-approach goal (PAPG), and performance-avoidance goal (PAVG). We excluded three 
items that had a factor loading of .40 or less; as a result, the above criterion was satisfied: CFI-pre = .963, 
RMSEA-pre = .047, CFI-post = .955, RMSEA-post = .061. Table 2 shows the Cronbach’s a coefficient and 
the items for each factor. The average value for the items was regarded as the respective value for each 
factor. 


Table 2. Items and o Coefficient 


Factors and items 

Learning goal (pre:a = .71, post: = .77) 

I want to learn as much as possible from classes. 

I hope to have gained a broader and deeper knowledge when I am done with classes. 
I prefer course material that really challenges me so I can learn new things. 

I prefer course material that arouses my curiosity, even if it is difficult to learn. 
Performance-approach goal (pre:a = .85, post:a = .86) 

It is important to me to do better than the other students. 

My goal is to get better grades than most of the students. 

I am striving to demonstrate my ability in relation to others. 

I am motivated by the thought of outperforming my peers. 

It is important to me to do well as compared to others. 

I want to do well to show my ability to my instructors, friends, and family. 
Performance-avoidance goal(pre:a = .73, post:a = .78) 

I worry about the possibility of getting a bad grade. 

I often think to myself, "What if I do badly?". 

I just want to avoid doing poorly. 

My fear of performing poorly is often what motivates me. 

I’m afraid that if I ask my instructor a “dumb” question, they might not think ?m very smart. 


3.2 Mixed-design Analysis of Variance 


A mixed-design analysis of variance (mixed-design ANOVA) of group (five groups) x time (pre, post) was 
performed. Table 3 shows the results of this ANOVA. A marginally significant main effect of time was 
found for LG, but no other significant effect was found. The results of the mixed-design ANOVA show no 
effect of intervention on goal orientation. 


Table 3. Results for Mixed-Design ANOVA 


Control Presentation Construction Presentation Construction Group Time Interac 
-Self -Self -Peer -Peer (df) (df) tion 
mean SD mean SD mean SD mean SD _ mean SD (df) 


LG-pre 3.82 0.71 3.83 0.67 3.85 055 3.90 043 4.25 0.37 1.24 3.26+ 0.68 
LG-post 3.91 0.77 4.08 0.70 3.83 0.40 4.04 0.41 4.29 0.52 (4) (1) (4) 
PAPG-pre 3.30 0.76 3.15 0.84 3.44 0.59 3.32 0.60 3.56 0.77 0.65 2.66 0.12 
PAPG-post 3.14 0.79 3.09 0.80 3.40 0.66 3.18 0.53 3.44 0.77 (4) (1) (4) 
PAVG-pre 3.90 0.64 3.55 0.84 3.83 0.44 3.83 0.52 3.93 0.64 0.80 2.02 1.03 
PAVG-post 3.54 0.84 3.51 1.03 3.87 0.45 3.67 0.41 3.95 0.55 (4) (1) (4) 


+<.10. 
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3.3 Cross-Lagged Model 


To examine the cross-lagged relationships among LG, PAPG, and PAVG between pre- and post-intervention, 
we hypothesized a cross-lagged model. SEM served as the basis for exploring the relationships among 
variables in the cross-lagged model. The results of an SEM analysis to determine whether the cross-lagged 
model fit was acceptable, the criterion was satisfied: CFI = 1.00, RMSEA = .00. 

Analysis 1: Multi-group structural equation modeling analysis 

We conducted a multi-group SEM analysis to determine whether the cross-lagged model fit was 
acceptable among the groups. We excluded the path from PAVG-pre to PAPG-post, which was not 
significant at the 10% level for any group; as a result, the criterion was satisfied: CFI = .995, RMSEA = .034. 

Figure | shows the cross-lagged model for each group. Determination coefficients, error variables, 
correlation paths and their path coefficients, and autoregressive paths (e.g., LG-pre — LG-post) and their 
path coefficients are omitted from Fig. 1 for clarity. The autoregressive path coefficients account for the 
stability of each answer between pre- and post-intervention. Most autoregressive path coefficients in all 
groups were significant at 5% level. The exceptions were the effects of PAVG-pre on PAVG-post 
(significant at the 10% level) in Control Group and the effect of PAVG-pre on PAVG-post in the 
Construction-Self Group (n. s.). 

The results of the cross-lagged effects are described below. In the Control Group and Presentation-Self 
Group, no significant cross-lagged effects were found. In the Construction-Self Group, PAPG-pre had a 
positive effect (§ = .65) on LG-post. In the Presentation-Peer Group, LG-pre had a positive effect (6 = .30) on 
PAPG-post; PAPG-pre had a positive effect (8 = .52) on LG-post; and PAVG-pre had a negative effect 
(B = -.46) on LG-post. In the Construction-Peer Group, LG-pre had a negative effect (8B = -.45) on 
PAVG-post; PAPG-pre had positive effects on LG-post (8B = .75) and PAVG-post (8B = .35); PAVG-pre had a 
negative effect (§ = -.94) on LG-post. 


re re 


| Le PAPG PAVG | Le PAPG PAVG 


Cis) 
(03) (41) @n (05) (-.20) 


(;.16) (+09) (.00) . (02) 


F z = =x x 3 2 <x — >= : 
| ue PAPG | | PAVG | ie | PAPG PAYG 
= post post 

a. Control Group b. Presentation-Self Group 
pre pre 
[ LG | PAPG PAVG | LG PAPG _—s PAVG 


(03) 
1.30) 


| LG PAPG | PAVG ie PAPG PAVG 
; post post 


c. Construction-Self Group d. Presentation-Peer Group 


Note. 
Values on the path are normalized coefficients 
(***p<.001,**p<.01,*p<.05,p<.10). 


Paths which are not significant at 10% level are indicated by 


broken lines, normalization factors are indicated in 


parentheses. 


post 


e. Construction-Peer Group 


Figure 1. Cross-lagged Model for Each Group 
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Analysis 2: Test for path—coefficient differences 

We tested for measurement invariance to determine whether path coefficients differed significantly 
between the control group and the four experimental groups. Figure 2 indicates whether the path coefficients 
in the four experiment groups differ significantly at the 10% level compared to the Control Group. 

No path coefficient exhibiting a significant difference between the Control Group and_ the 
Presentation-Self Group. The path coefficient from PAPG-pre to LG-post differed significantly between the 
Control Group and the Construction-Self Group (z = 1.82, p < .10). Two path coefficients from PAPG-pre to 
LG-post (z = 1.71, p < .10) and from the PAVG-pre to LG-post (z = 1.72, p < .10) differed significantly 
between the Control Group and the Presentation-Peer Group. Three paths from LG-pre to PAVG-post 
(z = 2.17, p < .05), from PAPG-pre to LG-post (z = 2.08, p < .05), and from PAVG-pre to LG-post 
(z = 2.62, p < .01) differed significantly between the Control Group and the Construction-Peer Group. 


pre pre 
LG | | PAPG PAVG | | LG PAPG PAVG | 
LG | | PAPG PAVG | | LG PAPG | PAVG | 
post post 
a. Presentation-Self Group b. Construction-Self Group 
pre pre 
| LG | PAPG PAVG | LG PAPG | | PAVG | 
, rae oor" i. ee 
+= - —————$—_—_—— = _ ae 
| LG | PAPG PAVG | PAPG | PAVG | 
post post 
c. Presentation-Peer Group d. Construction-Peer Group 


Note. Dashed line shows significant at 10% level, solid line shows significant at 5% level, thick solid line shows 
significant at 1% level. 


Figure 2. Paths Exhibiting Significant Difference 


4. DISCUSSION 


4.1 Effects of the Intervention 


This was a 2 x 2 factor study, measuring evaluation indicators (presentation/construction) and evaluator 
(self/peer) on goal orientation were manipulated. The results, assessed using mixed-design ANOVA, showed 
no salient features. The results of analysis 1 and 2, nevertheless, indicated that there were differences in the 
effects of intervention among the groups, and these effects were shown. These results showed that the effects 
of the intervention on goal orientation may depend on the individual student’s initial goal orientation. In 
other words, simple analyses of changes in goal-orientation scores before and after the intervention (Winne et 
al. 2003; Geitz et al. 2015) are insufficient for finding the effects of an intervention. Instead, it should be 
concluded that the effects of the intervention were affected by the individual student’s initial goal orientation. 
This study provides a new perspective for intervention studies that change goal orientation. 

In analysis 1, cross-lagged relationships for goal orientation were examined for each group. No 
significant cross-lagged effect was found in the Control Group or the Presentation-Self Group, while there 
was a significant cross-lagged effect for the Construction-Self Group, three significant cross-lagged effects in 
the Presentation-Peer Group, and four significant effects in the Construction-Peer Group. In analysis 2, we 
tested for measurement invariance to examine whether path coefficients significantly differed between the 
control group and each of the four experiment groups. There was no path coefficient found that exhibited a 
significant difference between the Control Group and the Presentation-Self Group, while there were path 


32 


International Conference Educational Technologies 2019 


coefficients that exhibited a significant difference between the Control Group and each of the other 
experimental groups (Construction-Self Group: one path coefficient; Presentation-Peer Group: two path 
coefficients; and Construction-Peer Group: three path coefficients). The results of analysis 1 and 2 suggest 
that by increasing the level of the intervention, students’ initial goal orientation might have a greater effect on 
the goal orientation that the student exhibited after the intervention. Because there was no difference between 
the Presentation-Self Group and the Control Group, self-evaluation using given evaluation indicators may not 
be sufficient for obtaining the effects of the intervention. 


4.2 The Cross-Lagged Relationships 


The results of this study give guidelines on the intervention according to student's initial goal orientation. We 
discuss this below in three ways. 

Performance-approach goal-pre — learning goal-post 

In the Construction-Self, Presentation-Peer, and Construction-Peer Groups, PAPG-pre had a greater 
positive effect on LG-post than the Control Group. Increasing levels of intervention for students who wanted 
to be evaluated positively (PAPG-pre) may promote their interest in understanding the contents and/or 
improving their skills (LG-post). It was suggested that it would be effective for students with a high 
performance-approach goal orientation to construct evaluation indicators or conduct peer-evaluations. 

Performance-avoidance goal-pre — learning goal-post 

In the Presentation-Peer Group and the Construction-Peer Group, PAVG-pre had a greater negative 
effect on LG-post than the Control Group. Because this effect was seen in the two Peer groups but not in the 
two Self groups, peer-evaluation may be the decisive factor here. Students who wanted to avoid being 
evaluated negatively (PAVG-pre) may have suppressed their interest in understanding content and/or 
improving their skills (LG-post) due to being evaluated by others. This result indicates that designing 
situations to receive evaluation from others may not be effective for students with a high 
performance-avoidance goal orientation. 

Learning Goal-pre — performance-avoidance goal-post 

Third, in the Construction-Peer Group, LG-pre had a greater negative effect on PAVG-post than in the 
Control Group. Students who were trying to understand the contents and improve their abilities (LG-pre) and 
were evaluated by their peers according to their own evaluation indicators were able to increase their 
understanding of the significance of evaluation and reduce their fear of being evaluated (PAVG-post). This 
suggests that for students with a high learning goal orientation, combining their own construction of 
evaluation indicators and peer-evaluations may be effective. 


5. FUTURE WORK 


This study investigated students’ goal orientation by manipulating evaluation indicators 
(presentation/construction) and evaluator (self/peer). The overall results indicated that the intervention had 
positive effects for students with a high learning goal orientation or a performance-approach goal orientation, 
but it had negative effects for students with a high performance-avoidance goal orientation. In the future, 
effective methods for students with a high performance-avoidance goal orientation should be carefully 
considered. 

The multiple goal perspective (Elliot 1999; Harackiewicz et al. 2002; Pintrich 2000) indicates that 
students could endorse different levels for multiple goals. For example, they could endorse high levels of 
both the performance-approach goal and the performance-avoidance goal at the same time. This then 
indicated another future task, namely, to explore the effects of intervention on students with multiple goals. 
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